Vehicle gesture recognition system and method

ABSTRACT

Embodiments of vehicle gesture recognition systems and methods are disclosed. An example vehicle gesture recognition system comprises a data interface configured for receiving 2d image data from a 2d sensor and/or from a portable device camera via a portable device interface. Additionally or alternatively, the data interface is configured for receiving gesture data indicating a gesture. A vehicle processing unit is configured for controlling user interfacing with a user interface based on the gestures recognized from the 2d image data and/or as indicated by the gesture data.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. Non-Provisional patentapplication Ser. No. 14/091,281, entitled “VEHICLE GESTURE RECOGNITIONSYSTEM AND METHOD,” and filed on Nov. 26, 2013. U.S. Non-Provisionalpatent application Ser. No. 14/091,281 claims priority to EuropeanPatent Application No. 12195018.2, filed on Nov. 30, 2012. The contentsof each of the above-referenced applications are hereby incorporated byreference for all purposes.

TECHNICAL FIELD

Various embodiments relate to a vehicle gesture recognition system and amethod of gesture recognition in a vehicle. In particular, variousembodiments relate to techniques which allow for receivingtwo-dimensional image data for gesture recognition and/or for receivingof gesture data indicating a gesture.

BACKGROUND

Techniques for gesture recognition are often utilized in the field ofuser input/output and user interfacing. For example, a gesture mayrelate to a user using a part of the body, e.g., a hand or fingers ofthe hand, for non-verbal communication. Other examples of gestures arehead movements, body movements, sign language, hand and arm movement,etc. Techniques may be used to recognize the gesture and use the gestureto control a user interface. In vehicles, gesture recognition may allowfor interfacing with a vehicle head unit which controls differentfunctionalities of the vehicle.

Typically, for gesture recognition, a plurality of three-dimensional(3d) cameras or stereoscopic cameras indicating a distance of a certainobject with respect to the at least one camera, are employed. From thevarious perspectives, e.g., by means of a stereoscopic calculation,and/or the 3d information, reliable gesture recognition is possible.Such techniques typically rely on determining a 3d position of agesture-related object, e.g. a user's hand.

However, such vehicle gesture recognition systems may face certainrestrictions. For example, the different parts may be comparably costly.Moreover, the control and implementation of such systems may becomparably complex and time-consuming. Training of the system may beneeded. Failures may be comparably likely to occur due to the high levelof complexity.

In particular in the vehicle environment, multiple ways of userinterfacing are available, including buttons, touch panels, rotarybuttons, etc. Therefore, the additional high costs of a vehicle gesturerecognition system may be undesired.

SUMMARY

According to an aspect, a vehicle gesture recognition system isprovided. An example vehicle gesture recognition system comprises a datainterface configured for receiving two-dimensional (2d) image data of atleast one field of view within the vehicle. The vehicle gesturerecognition system further comprises a vehicle processing unitconfigured for recognizing a gesture from the received 2d image data andcontrolling user interfacing with a user interface based on therecognized gesture.

The 2d image data may relate to a 2d array of pixels where each pixelhas at least one assigned pixel value. For a 2d camera operative invisible range, the pixel value may be representative of a brightnessand/or color. In contrast to 3d image data, the pixel value may not bedirectly representative of a distance from the imaged object to thecamera. In other words, it may be dispensable to determine a 3d positionof a gesture-related object, e.g. a user's hand. It may be sufficient todetermine a 2d position of the gesture-related object. The 2d image datamay be provided in conventional data formats, such as still pictureformats, e.g. jpg, tiff, etc., or image stream formats, moving pictureformats, etc. For example, it is possible that the 2d image data imagesa time series of a given length in order to detect a dynamic gesture,e.g. a movement or the like. It is possible that the 2d image datacomprises vectors indicating a direction of a dynamic movement.

By using 2d image data, comparably simple gesture recognition may beemployed—in particular if compared to more complex systems, where 3dimage data is used for the gesture recognition. In particular, gesturerecognition may be employed which is able to recognize gestures selectedfrom the group comprising gestures such as: wipe left-to-right, wiperight-to-left, wipe top-to-bottom, wipe bottom-to-top, turn right, turnleft, push, handwriting, sign language, etc.

For example, the push gesture may be recognized based on an objectedrelated to the gesture becoming larger over a time-series of the 2dimage data. Respectively, a wipe left-to-right gesture may be recognizedbased on the object moving from one side to the other side in thetime-series of the 2d image data.

For example, it is possible to restrict the gesture recognition to apredefined set of gestures selected from the group as described above.Typically, such gestures as described with respect to the group abovecan be used to control the user interfacing with a graphical userinterface (GUI) as often present in vehicles. For example, the wipecommands can be used for cursor control, while the turning commands canbe used for dialing commands, and the pushing gesture can be used forexecuting commands. Such examples of possible applications of therecognized gesture in the user interfacing are merely illustrating andshall not be construed as being limiting.

For example, by restricting said recognizing of the gesture to apredefined set of gestures as discussed above, an increased likelihoodof successful gesture recognition may be achieved. Moreover, byemploying the 2d image data for said recognizing of the gesture,simplified processing as part of said recognizing of the gesture may beachieved. Fewer computational resources occupied at the vehicleprocessing unit and shorter latency times may be achieved.

User interfacing may relate to the interaction of a user with the userinterface. The interaction may comprise output to the user and/or inputfrom the user. Typical examples for the user interface are GUIs where amenu structure or the like, together with information is graphicallypresented to a user of the system.

The field of view may relate to a line of sight or sensitive region ofan image source of the 2d image data. In other words, the field of viewmay define the region which is covered by the 2d image data. Therefore,when a user executes the gesture within the field of view, the 2d imagedata may image or picture the gesture.

For example, the data interface may receive the 2d image data from anexternal image source. The external image source, e.g., a third-partyportable device such as a webcam, cell phone, gaming console, touchpad,or a laptop, etc. may provide the 2d image data to the vehicle gesturerecognition system. It is possible that the external image source is atouch pad. In such a scenario the field of view may refer to thesensitive region of the touch pad. By such techniques, it may bedispensable to provide a dedicated image source for acquiring the 2dimage data—yet it is possible to provide an image source.

The vehicle gesture recognition system may comprise a portable deviceinterface configured for establishing a data connection between aportable device and the data interface. The data interface may beconfigured to receive at least parts of the 2d image data via the dataconnection from a 2d sensor of the portable device.

For example, the portable device may be selected from the groupcomprising: a webcam, a cell phone, a gaming console, a touchpad, and alaptop. For example, the 2d sensor may be a 2d optical camera or a touchpad.

For example, the data connection between the portable device and thedata interface may be established by the portable device interface usingone or more data connection standards selected from the groupcomprising: Wireless Local Area Network (WLAN), Bluetooth, Near FieldCommunication (NFC), Ethernet, Universal Asynchronous ReceiverTransmitter (UART), Media Orientated System Transport (MOST), ControllerArea Network (CAN), Universal Serial Bus (USB). The data connection mayemploy a wired link and/or a wireless link. Different parts of the dataconnection, e.g. between the data interface and the portable deviceinterface, as well as between the portable device interface and theportable device, may operate according to different data communicationstandards, etc.

Besides such electronic functionality as described above, the portabledevice interface may also provide mechanical functionality.

In particular, the portable device interface may be configured toreleasably mount the portable device in a predetermined position.

For example, the portable device interface may be arranged in awell-defined position within a reference coordinate system explicitly orimplicitly known by the vehicle gesture recognition system. Thereby, thefield of view of the 2d sensor or camera of the portable device may havea well-defined and predetermined arrangement as well. This mayfacilitate said recognizing of the gesture, because the sensor or cameraposition may be known.

For example, it is possible that the portable device interface receives,holds in a fixed position, and—upon user actuation—releases the portabledevice. In other words, the portable device interface may be configuredfor releasably mounting the portable device.

For example, the predetermined position of the portable device interfacemay be achieved by fixedly mounting the portable device interface to aparticular position in the interior of the vehicle. In a state where theportable device is mounted to the portable device interface, it may thenbe possible to determine the position of the portable device as well.

For example, the portable device interface may further comprise lockingmeans which are configured for engaging with side surfaces of theportable device for said releasably mounting. The locking means maycomprise resilient means which secure the portable device and may beactuated in order to release the portable device from the portabledevice interface.

It should be understood that it is also possible that the portabledevice interface does not comprise any specific mechanical parts.Rather, it may be employed in a wireless manner where the portabledevice can be connected to the portable device interface and, at thesame time, move around freely, e.g., in the entire vehicle interior.

When employing the portable device to acquire and provide the 2d imagedata to the data interface via the portable device interface, it may bepossible to reduce the number of parts and units dedicated to thegesture recognition. For example, it is possible that the vehicleprocessing unit is a vehicle head unit or vehicle infotainmentdevice/head unit. Typically, vehicles may already comprise such avehicle head unit and the recognizing of the gesture can then beimplemented by a software code based on the 2d image data received fromthe portable device 2d sensor. In such a case, no additional hardwareelements may be necessary. This may reduce the costs for the vehiclegesture recognition system.

The vehicle gesture recognition system may further comprise at least one2d sensor configured for providing at least parts of the 2d image datato the data interface.

In other words, it may be possible to provide a dedicated 2d sensor. The2d sensor may be configured to acquire the 2d image data and thereby actas an image source of the 2d image data.

The at least one 2d sensor may be selected from the group comprising:infrared sensor; 2d optical camera; capacitive sensor; magnetic fieldsensor; ultrasonic sensor; and touch pad.

Typically, such 2d sensors may have a comparably limited spatialresolution, e.g., less than 1000×1000 pixels, or less than 500×500pixels, and may, therefore, be comparably inexpensive. It may thereforebe possible to reduce the costs of the overall vehicle gesturerecognition system.

For example, when a gesture-related object, e.g. a user's hand, moves infree space within the field of view of the magnetic field sensor(capacitive sensor), a distribution of the magnetic field (electricfield) may get distorted which can be sensed by the sensor in aspatially resolved manner.

For example, when the gesture-related object moves above or in contactwith the touch pad, this may be sensed by the sensor in a spatiallyresolved manner.

For example, it is possible that the vehicle gesture recognition systemcomprises a plurality of image sources, e.g., one or more 2d sensorsand/or the portable device interface for receiving parts of the 2d imagefrom the at least one 2d sensors and other parts of the 2d image datafrom the portable device 2d sensor. In such a case, said recognizing ofthe gesture may be executed for all or parts of the 2d image datareceived from the various image sources. For example, if the fields ofview associated with the different image sources overlap, it may bepossible to recognize said gesture in each of the respective 2d imagedata. One possible technique is to recognize said gesture individuallyfrom the respective parts of the 2d image data corresponding to thevarious image sources.

Moreover, because it may be possible that the fields of view of thedifferent 2d image data differ, e.g. have different arrangement inspace, the different 2d image data may correspond to variousperspectives of the gesture. It may be possible to make use of thevarious points of views or perspectives of the gesture when recognizingthe gesture from the received 2d image data. This may be achieved byrecognizing said gesture in a correlated manner by taking into accountthe respective parts of the 2d image data corresponding to the variousimage sources in a combined and inter-coordinated gesture recognition.Examples may include stereoscopic reconstruction from the differentperspectives of the gesture. Thereby, an increased likelihood ofsuccessfully recognizing the gesture may be achieved.

The vehicle processing unit may be a vehicle head unit, the vehicle headunit comprising a housing. At least one outer surface of the housing maybe a cover shielding an interior of the vehicle head unit from apassenger compartment of the vehicle. The cover may comprise interfaceelements. The at least one 2d sensor may be located on the at last oneouter surface.

The vehicle head unit may refer to a system of various electricalcomponents which are integrated and arranged within the housing andprovide various functionality such as vehicle control, air conditioningcontrol, entertainment, communication and connectivity, safety andsecurity, driver assistance, etc. The vehicle head unit may provide agraphical user interface (GUI) for such functionality. Typically, thevehicle head unit may be arranged close to the center console of thevehicle. The cover may form a part of the vehicle center console. Theinterface elements, e.g. buttons, rotary push buttons, a touch panel,etc., may allow user interfacing with the GUI. Said user interfacingbased on the gesture may allow controlling the GUI to access informationand/or execute commands.

In other words, the at least one 2d sensor may be provided physicallyintegrated with the vehicle head unit. This may allow for simplifiedconnection between the 2d sensor and the data interface. The 2d sensormay be provided in the vehicle center console.

The field of view of the at least one 2d sensor may be orientatedsubstantially perpendicular to the at least one outer surface. In otherwords, the field of view may be orientated towards a central arm restand/or cover a central area between the two front seats of the vehicle.In such a scenario, it may be possible to reliably detect gesturesperformed by a driver's right hand in front of the center console.

The portable device interface may be configured to releasably mount theportable device such that the field of view of the at least one 2dsensor and the field of view of the portable device 2d sensor are atleast partially overlapping.

In other words, in a scenario where there is provided the at least one2d sensor and the portable device interface coupled to the portabledevice 2d sensor, the respective fields of view may be arranged suchthat they cover a common region. If the gesture is executed in thatcommon region, it may be recognized from the 2d image data obtainedfrom, both, the portable device 2d sensor and the at least one 2dsensor. This may increase the likelihood of successful gesturerecognition.

For example, it is possible that the fields of view of the portabledevice 2d sensor (as defined by the portable device interface) and theat least one 2d sensor enclose a certain angle with each other. Forexample, this certain angle may amount to 90°±45°, preferably 90°±20°.

In particular, it may be possible that the vehicle processing unit isconfigured for recognizing the gesture taking into account thepredetermined position of the portable device and/or a predeterminedposition of the at least one 2d sensor.

For example, when the predetermined positions of the at least one 2dsensor and the portable device 2d sensor are known, e.g., with respectto each other or relative to each other, this may allow for astereoscopic reconstruction of the gesture. By such means, athree-dimensional (3d) movement may be reconstructed from the 2d imagedata which may increase the likelihood of successful recognition of thegesture. Yet, it should be understood that in various simple scenariosthe gesture may be individually detected for each of the at least one 2dsensor and the portable device 2d sensor, i.e., without determining any3d coordinates.

The vehicle processing unit may be further configured for determiningthe predetermined position of the portable device based on the parts ofthe 2d image data received from the portable device 2d sensor andfurther based on parts of the 2d image data received from the at leastone 2d sensor.

In other words, when the fields of view of the different parts of the 2dimage data differ, it may be possible to deduce the camera position ofthe portable device camera, e.g. from a stereoscopic reconstruction ofthe recognized gesture. It may be possible to prompt the user to executea well-defined calibration gesture for this purpose. Such a scenario mayhave the effect of flexible positioning of the portable device withinthe vehicle interior—while, at the same time, preserving a highrecognition likelihood by using 2d image data from a plurality of 2dimage data source. Such techniques may be particularly desirable whenthe portable device interface employs a wireless data connection to theportable device which allows a user to freely position the portabledevice according to his/her needs.

In the techniques discussed above, it may be possible to recognize thegesture based on the 2d image data. A certain limited, yet finite amountof computational resources may be required for said recognizing.

According to a further aspect, a vehicle gesture recognition system isprovided comprising a data interface configured for receiving gesturedata, the gesture data indicating a gesture. The vehicle gesturerecognition system further comprises a portable device interface beingconfigured for establishing a data connection between a portable deviceand the data interface. The data interface is further configured forreceiving the gesture data via the data connection from a portabledevice application executed on the portable device. The vehicleprocessing unit is configured for controlling user interfacing with auser interface based on the gesture indicated by the gesture data.

The vehicle gesture recognition system according to the presentlydiscussed aspect may have the effect that even less computationalresources may be required. In particular, it may be possible topre-recognize the gesture and merely receive the gesture data which,e.g. in parameterized form, indicates the gesture. In such scenarios,the vehicle gesture recognition system according to the presentlydiscussed aspect may use the pre-recognized gesture for said userinterfacing. As the vehicle gesture recognition system according to thepresent aspect may operate based on the gesture data, it may, moregenerally, also be referred to as a gesture controlled vehicle userinterface.

The vehicle gesture recognition system may further comprise a portabledevice which is configured for being connected to the portable deviceinterface. The portable device may comprise a touch sensitive display.The portable device may be configured to recognize a touch gesture bymeans of the touch sensitive display. The portable device may further beconfigured to determine gesture data, the gesture data indicating thetouch gesture. Optionally, the portable device may further comprise aninterface configured for sending the gesture data to a vehicle gesturerecognition system according to further aspects of the disclosure.

For example, the gesture data may comprise indicators indicating atleast one particular gesture from a predefined set of gestures. Theindicators may indicate the gesture implicitly or explicitly. A look uptable may be employed.

In other words, the gesture data may indicate one or more gestures,e.g., indirectly by means of a code—lookup table combination or thelike. For example, the lookup table may correspond to the predefined setof gestures. The computational resources required at the vehicleprocessing unit may be limited to said user interfacing based on thepre-recognized gesture indicated by the gesture data.

In other scenarios, the gesture may be identified by the indicators byspecifying a start point and/or an end point of a linear movement. Inyet further scenarios, the gesture may be identified by the indicatorsby specifying a series of trace points of an arbitrary movement. Suchpoints may relate to 2d or 3d coordinates of the movement in space. Infurther scenarios, the indicators can alternatively or additionallyindicate a velocity of the movement associated with the gesture.

It is possible that the vehicle gesture recognition system according tothe presently discussed aspect is configured according to a vehiclegesture recognition system according to further aspects of the presentdisclosure. In particular, it is possible that the data interface is,furthermore, configured for receiving the 2d image data and that thevehicle processing unit is, furthermore, configured for recognizing afurther gesture from the received 2d image data.

There are various possibilities how the gesture data is determined bythe portable device. In one scenario, the portable device may beconfigured to acquire 2d image data by means of a portable devicecamera. The portable device may be further configured to recognize thegesture from the acquired 2d image data and determine the gesture datain response to and based on said recognizing.

In a further scenario, the portable device may be configured torecognize a touch gesture by means of a touch sensitive display (touchpanel) of the portable device. The touch gesture may relate to a contactor close-contact of an object, e.g. a user's hand, with the touch panel.The portable device may be further configured to determine the gesturedata in response to and based on said recognizing. In such a scenariomore complex gestures may be recognized, e.g. handwriting. It may bepossible to control the user interface based on such gestures to enter aname, a song title, or a navigation destination.

According to a further aspect, a method of gesture recognition in avehicle is provided. The method comprises receiving 2d image data of atleast one field of view within the vehicle. The method furthercomprises, in a vehicle processing unit, recognizing a gesture from thereceived 2d image data. The method further comprises, in the vehicleprocessing unit, controlling user interfacing with a user interfacebased on the recognized gesture.

For such a method of gesture recognition, effects may be obtained whichare comparable to the effects which may be obtained with a vehiclegesture recognition system according to a further aspect of the presentdisclosure.

According to a further aspect, a method of gesture recognition in avehicle is provided. The method comprises establishing a data connectionbetween a portable device and a portable device interface and receivinggesture data indicating a gesture via the data connection from aportable device application executed on the portable device. The methodcomprises, in a vehicle processing unit, controlling user interfacingwith a user interface based on the gesture indicated by the gesturedata.

For such a method of gesture recognition, effects may be obtained whichare comparable to the effects which may be obtained with a vehiclegesture recognition system according to a further aspect of the presentdisclosure.

According to a further aspect, a vehicle gesture recognition system isprovided. The vehicle gesture recognition system comprises a datainterface configured for receiving 2d image data of at least one fieldof view within the vehicle. The data interface is further configured forreceiving gesture data indicating a gesture. The vehicle gesturerecognition system further comprises a vehicle processing unitconfigured for selective recognizing a further gesture from the 2d imagedata, in response to said receiving of the 2d image data. The vehiclegesture recognition system is further configured for controlling userinterfacing with a user interface based on the gesture indicated by thegesture data and/or based on the recognized further gesture.

According to a further aspect, a portable device gesture recognitionsystem is provided. The portable device gesture recognition systemcomprises a 2d camera with a field of view. The portable device isconfigured to acquire 2d image data by means of the 2d camera and torecognize a gesture from the acquired 2d image data. The portable deviceis further configured to determine gesture data, the gesture dataindicating the gesture. Optionally, the portable device may furthercomprise an interface configured for sending the gesture data to avehicle gesture recognition system according to further aspects of thedisclosure.

According to a further aspect, a portable device gesture recognitionsystem comprising a portable device is provided. The portable devicecomprises a touch sensitive display. The portable device is configuredto recognize a touch gesture by means of the touch sensitive display.The portable device is further configured to determine gesture data, thegesture data indicating the touch gesture. Optionally, the portabledevice may further comprise an interface configured for sending thegesture data to a vehicle gesture recognition system according tofurther aspects of the disclosure.

According to a further aspect, a vehicle gesture recognition system isprovided which comprises a data interface. The data interface isconfigured to receive two-dimensional, 2d, image data of at least onefield of view within the vehicle. The vehicle gesture recognition systemfurther comprises a first 2d sensor configured for providing at leastparts of the 2d image data to the data interface. The vehicle gesturerecognition system further comprises a second 2d sensor configured forproviding at least parts of the 2d image data to the data interface. Afield of view of the first 2d sensor is at least partially overlappingwith a field of view of the second 2d sensor. The fields of view of thefirst and second 2d sensors are approximately perpendicular with respectto each other. The vehicle gesture recognition system further comprisesa vehicle processing unit configured for recognizing a gesture from thereceived 2d image data, and further configured for controlling userinterfacing with a user interface based on the recognized gesture.

Because the fields of view of the first and second 2d sensors encloseapproximately an angle of 90°, i.e., an angle of 90°±20°, moreadvantageously 90°±5°, even more advantageously 90°±1°, they can depictthe gesture from different and complementary view points. Thereby, athree-dimensional gesture may be recognized. Further, the recognizingmay be more reliable.

For example, the first 2d sensor may be selected from the groupcomprising: infrared sensor; 2d optical camera; capacitive sensor;magnetic field sensor; ultrasonic sensor; and touch pad. For example,the second 2d sensor may be selected from the group comprising: infraredsensor; 2d optical camera; capacitive sensor; magnetic field sensor;ultrasonic sensor; and touch pad.

The first sensor and/or the second sensor may be fixedly integrated withthe vehicle.

It is to be understood that the features mentioned above and featuresyet to be explained below can be used not only in the respectivecombinations indicated, but also in other combinations or in isolation,without departing from the scope of the present disclosure. Features ofthe above-mentioned aspects and embodiments may be combined with eachother in other embodiments. In particular, the features of the variousvehicle gesture recognition systems of different aspects may becombined. It may also be possible that the aspects of the portabledevice gesture recognition systems are combined with the vehicle gesturerecognition system, e.g., in one system.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following, the disclosure will be explained in further detailwith respect to embodiments illustrated in the accompanying drawings.

FIG. 1 is a schematic illustration of a vehicle gesture recognitionsystem according to various embodiments;

FIG. 2 illustrates a gesture;

FIG. 3 illustrates gesture data including indicators indicating at leastone specific gesture from a predefined set of gestures;

FIG. 4 is a perspective view of a vehicle center console including thevehicle gesture recognition system of FIG. 1;

FIG. 5 illustrates fields of view of a 2d sensor and a cell phone camerafor the system of FIG. 4; and

FIG. 6 is a flowchart of a method of gesture recognition in a vehicleaccording to various embodiments.

DETAILED DESCRIPTION

The foregoing and additional features and effects of the disclosure willbecome apparent from the following detailed description when read inconjunction with the accompanying drawings, in which like referencenumerals refer to like elements. The drawings are to be regarded asbeing schematic representations, and elements illustrated in thedrawings are not necessarily shown to scale. Rather, the variouselements are represented such that their function and general purposebecome apparent to a person skilled in the art. Any connection orcoupling between functional blocks, devices, components or otherphysical or functional units shown in the drawings or described hereinmay also be implemented by an indirect connection or coupling. Acoupling between components may also be established over a wirelessconnection. Functional blocks may be implemented in hardware, firmware,software or a combination thereof.

In FIG. 1, a vehicle gesture recognition system 100 is schematicallyillustrated. The vehicle gesture recognition system 100 comprises avehicle processing unit 120 coupled to interface elements 122 and adisplay 150. For example, the interface elements 122, sometimes referredto as control elements, can comprise a touch panel, buttons, rotary pushbuttons, etc. The vehicle processing unit 120 is configured to provide aGUI on the display 150 for user interfacing.

Furthermore, the vehicle processing unit is coupled to a data interface110. The data interface 110 can be configured to receive 2d image datafrom an at least one image source covering a field of view within thevehicle; additionally or alternatively the data interface 110 can beconfigured to receive gesture data indicating a gesture. For example,when the data interface 110 is configured for receiving 2d image data,the vehicle processing unit 120 can recognize a gesture from thereceived 2d image data.

Turning to FIG. 2 where a gesture 300 is illustrated. For example, thedepicted hand of a user may be wiped horizontally or vertically or maybe turned. These movements may correspond to different gestures.

In FIG. 3, gesture data 310 is schematically illustrated. The gesturedata 310 comprises two indicators 311, each indicating a particulargesture 300 in a parameterized manner. For example, the gesture labeled“7” may correspond to a wiping gesture from left to right, while thegesture labeled “3” may correspond to a turning of the hand. Suchgestures 300 as discussed with respect to FIG. 3 above are merelyillustrative and are not to be construed as being limiting.

Turning back to FIG. 1, by recognizing the gesture 300 from the 2d imagedata and/or by obtaining the pre-recognized gesture 300 from theindicators 311 of the gesture data 310, a certain gesture 300 isprovided to the vehicle processing unit 120. Based on this gesture 300,the vehicle processing unit 120 is configured for controlling userinterfacing with the GUI displayed on the display 150. As a specificexample, it is possible that a wiping gesture from left to right moves acursor of the GUI. Respectively, it is possible that a pushing gestureexecutes a certain command or menu entry of the GUI where the cursor iscurrently positioned.

As will be appreciated from the above, the gesture control of the GUIcan be based on either 2d image data of the gesture 300, or the gesture300 indicated by the gesture data 310, or a combination thereof.Scenarios where the 2d image data is employed will be discussed next.

For example, the data interface 110 may receive at least parts of the 2dimage data from a 2d sensor 140. For example, the 2d sensor 140 can bean infrared sensor, a 2d optical camera, a capacitive sensor, a magneticfield sensor, or an ultrasonic sensor. It is possible to provide aplurality of 2d sensors 140 which can each contribute to providing the2d image data. In such a scenario, each of the plurality of 2d sensors140 may provide a part of the 2d image data.

In particular, compared to scenarios where 3d sensors and 3d image datais provided, said recognizing of the gesture from the received 2d imagedata may be comparably simple. This may reduce the requiredcomputational efforts for said recognizing in the vehicle processingunit 120. At the same time, it is possible to provide a reliablerecognizing of the gesture 300, e.g. by providing a plurality of 2dsensors 140. Redundant and multiple recognizing of the gesture 300 maythen be possible, e.g. if the respective fields of view of the pluralityof 2d sensors 140 overlap and the gesture is executed in the commonregion.

In further scenarios, a cell phone interface 130 is configured forestablishing a data connection between a cell phone (not shown in FIG.2) and the data interface 110. In general, a connection may beestablished between the portable device interface 130 to any portabledevice, e.g., a touch pad, laptop, gaming console, webcam, cell phoneetc. The data interface 110 is configured to receive at least parts ofthe 2d image data via the data connection of the cell phone interface130 from a 2d camera of the cell phone. In other words, it is possibleto employ the 2d camera of the cell phone as image source in order toacquire and receive the 2d image data.

It is possible to receive the 2d image data via the cell phone interface130 alternatively or additionally to 2d image data received from atleast one 2d sensor 140. In other words, in various examples it may bedispensable to provide the 2d sensor 140; in other scenarios, it may bedispensable to provide the cell phone interface 130.

Additionally or alternatively to the receiving of at least parts of the2d image data via the cell phone interface 130, the cell phone interface130 can be configured to receive the gesture data 310. This will beexplained in more detail below.

In one scenario, the gesture data 310 relates to a gesture 300 executedwithin the field of view of a cell phone camera. An application executedon the cell phone can recognize this free-space gesture 300, e.g. a wipeleft-to-right or turning of the hand, and determine the respectivegesture data 310 for the recognized gesture 300.

In a further scenario, the gesture data 310 relates to a gesture 300executed on a touch panel of the cell phone. Cell phones may comprise atouch-sensitive display as the touch panel. Typically, capacitivesensors are employed for this purpose. An application executed on thecell phone can recognize this touch gesture 300 and determine therespective gesture data 310 for the recognized gesture 300. It ispossible to obtain the gesture data from both, a cell phone touch paneland a cell phone camera.

In other words, the cell phone interface 130 can be configured toreceive 2d image data and/or gesture data 310. Depending on theparticular application, i.e. if it is desired to recognize free-spacegestures 300 executed in a field of view of the cell phone camera and/orrecognizing touch gestures 300 executed on a touch panel of the cellphone, either one of the scenarios or both scenarios may be implemented.

The units illustrated in FIG. 1 can be implemented using hardware and/orsoftware where applicable. For example, it is possible to implement thedata interface 110 and/or cell phone interface 130 primarily hassoftware executed on the vehicle processing unit. This may be inparticularly true if wireless technology is applied for the dataconnection with the cell phone. Antennas for the wireless connection maybe provided.

Turning to FIG. 4, a perspective view of a center console and middleconsole of a vehicle is depicted. Visible in FIG. 4 is the display 150,the 2d sensor 140, the interface elements 122, and the cell phoneinterface 130. In the scenario depicted in FIG. 4, the vehicleprocessing unit 120 is associated with a vehicle head unit comprising ahousing where one outer surface is a cover 121 shielding an interior ofthe vehicle head unit from the passenger compartment of the vehicle. Theinterface elements 122, as well as the display 150, and the 2d sensor140 are located on the cover 121. In other words, the 2d sensor 140 isfully integrated with the vehicle head unit which makes the coupling anddata communication between the 2d sensor 140 and the data interface 110comparably simple.

As can be further seen from FIG. 4, the cell phone interface 130 isconfigured to releasably mount the cell phone in a predeterminedposition, namely in the scenario of FIG. 4, in the middle consolebetween driver and co-driver. The cell phone interface 130 is arrangedsuch that the mounted cell phone faces approximately along the verticaldirection. The cell phone interface 130 further comprises locking means,which engage with side surfaces of the cell phone in order to secure thecell phone in the predetermined position. In the scenario of FIG. 4 itis possible to obtain 2d image data from both, the 2d camera of the cellphone mounted to the cell phone interface 130 as well as from the 2dsensor 140 located at the outside surface 121 of the vehicle head unit120.

Turning to FIG. 5, the fields of view 160, 161 of the cell phone cameraand the 2d sensor 140, respectively, are graphically indicated. As canbe seen from FIG. 5, the cell phone interface 130 is configured forreleasably mounting the cell phone such that the fields of view 160, 161of the at least one 2d sensor and the cell phone camera overlap. Thefield of view 160 of the cell phone camera, i.e. as defined by the cellphone interface 130, is approximately orientated vertically. The fieldof view 161 of the 2d sensor 140 is approximately orientatedhorizontally. In particular, a common region of the fields of view 160,161 is located in the free space between the middle and center consoles.If the user executes the gesture 300 in this common region, the 2d imagedata received from both, the cell phone interface 130 and the 2d sensor140, depict the gesture 300.

The fields of enclose approximately an angle of 90°, i.e. offersubstantially different perspectives onto the common region. This mayfacilitate said recognizing of the gesture. Complementary 2d image datamay be available allowing to unambiguously recognize a wide variety ofgestures 300.

The vehicle processing unit 120 can then recognize the gesture 300individually from both 2d image data sets or can use knowledge about therelative positioning of the 2d sensor 140 with respect to the cell phonecamera in order to recognize the gesture 300 in a correlated manner. Inother words, the vehicle processing unit 120 can be configured torecognize the gesture 300 taking into account the predetermined positionof the cell phone and/or the predetermined position of the 2d sensor140. For example, this can occur as part of a stereoscopicreconstruction which allows to determine 3d position information of thegesture 300 executed within the field of views 160, 161. Respectivetechniques are known to the skilled person such that further detailsneed not be discussed in this context.

In the scenario of FIGS. 4 and 5, the cell phone interface 130 is shownin a fixed location on the middle console of the vehicle. Yet, it shouldbe understood that in various scenarios it is possible that the cellphone interface 130 is freely movable throughout the interior of thevehicle. For example, this can be the case when the data connectionbetween the cell phone interface 130 and the cell phone is establishedin a wireless manner and/or if the cell phone interface 130 isimplemented via software. In such a scenario it may be possible that thevehicle processing unit 120 is further configured for determining thepredetermined position of the cell phone—and therefore of the cell phoneinterface 130—based on parts of the 2d image data received from the 2dcamera of the cell phone and further based on parts of the 2d image datareceived from the at least one 2d sensor. For example, this may occur aspart of a correlated gesture recognition where based on a stereoscopicreconstruction, e.g. of a reference gesture 300, the camera position ofthe cell phone camera is determined.

In FIG. 6 a flowchart of a method of gesture recognition is shown. Themethod starts with step S1.

In step S2, the gesture data 310 is received. Step S2 is an optionalstep. As discussed previously with respect to FIG. 1, in variousscenarios no gesture data 310 is received—alternatively or additionallyto step S2, it is possible to receive 2d image data (step S3).

For example, the 2d image data in step S3 can be received from the 2dsenor 140 and/or the cell phone camera via the cell phone interface 130.

In step S4, it is checked whether 2d image data has been received instep S3. If so, in step S5, the gesture 300 is recognized from the 2dimage data. Otherwise, the method commences with step S6.

In step S6, the vehicle head unit 120 controls the GUI displayed on thedisplay 150 based on the gesture 300 which was recognized based on the2d image data in step S5 and/or indicated by the received gesture data310 in step S2.

The method ends in step S7. It should be understood that the particularorder of the steps may vary. E.g. it is possible to receive gesture datain step S4 only after step S5 has been executed.

As can be seen from the above, the vehicle head unit may rely on variousinput channels for receiving either pre-recognized gesture data and/or2d image data which can be used for gesture recognition. The 2d imagedata may originate from various image sources, e.g. cameras, touch pads,etc.

Although the disclosure has been shown and described with respect tocertain preferred embodiments, equivalents and modifications will occurto others skilled in the art upon the reading and understanding of thespecification. The present disclosure includes all such equivalents andmodifications. For example, while the disclosure has been describedpredominantly with respect to vehicles such as passenger cars, it shouldbe understood that vehicles such as airplanes, trains, trucks, etc. mayalso employ techniques described herein.

1. A vehicle gesture recognition system, comprising: a data interfaceconfigured for receiving gesture data, the gesture data including anindicator indicating a gesture from a predefined set of gestures, thegesture data including gestures detected based on image data from a 2dsensor of a portable device; and a portable device interface beingconfigured for: establishing a data connection between the portabledevice and the data interface, wherein the data interface is furtherconfigured for receiving the gesture data via the data connection from aportable device application executed on the portable device; and avehicle processing unit configured for controlling user interfacing witha user interface based on the gesture indicated by the gesture data. 2.The vehicle gesture recognition system of claim 1, wherein the indicatorindicates the gesture indirectly by use of a code and a lookup table,the lookup table corresponding to the predefined set of gestures.
 3. Thevehicle gesture recognition system of claim 1, wherein the indicatorindicates the gesture by specifying a start point of a linear movementassociated with the gesture.
 4. The vehicle gesture recognition systemof claim 1, wherein the indicator indicates the gesture by specifying anend point of a linear movement associated with the gesture.
 5. Thevehicle gesture recognition system of claim 1, wherein the indicatorindicates the gesture by specifying a series of trace points of amovement associated with the gesture.
 6. The vehicle gesture recognitionsystem of claim 5, wherein the series of trace points relate to 2D or 3Dcoordinates of the movement in space.
 7. The vehicle gesture recognitionsystem of claim 1, wherein the indicator indicates a velocity of amovement associated with the gesture.
 8. The vehicle gesture recognitionsystem of claim 1, wherein the at least one other 2d sensor isphysically integrated with a vehicle head unit in a vehicle.
 9. Thevehicle gesture recognition system of claim 1, wherein the fields ofview of the 2d sensor of the portable device and the at least one other2d sensor overlap in a common region located in free space of apassenger compartment between a middle console and a center console. 10.A method of gesture recognition in a vehicle, comprising: establishing adata connection between a portable device and a portable deviceinterface; receiving gesture data including an indicator indicating agesture from a predefined set of gestures via the data connection from aportable device application executed on the portable device, the gesturedata including gestures detected based on image data from a 2d sensor ofthe portable device; and in a vehicle processing unit, controlling auser interface based on the gesture indicated by the gesture data. 11.The method of claim 10, wherein the indicator indicates the gestureindirectly by use of a code and a lookup table, the lookup tablecorresponding to the predefined set of gestures.
 12. The method of claim10, further comprising identifying the gesture by the indicator, theindicator specifying a start point of a linear movement associated withthe gesture.
 13. The method of claim 10, further comprising identifyingthe gesture by the indicator, the indicator specifying an end point of alinear movement associated with the gesture.
 14. The method of claim 10,further comprising identifying the gesture by the indicator, theindicator specifying a series of trace points of a movement associatedwith the gesture.
 15. The method of claim 14, wherein the series oftrace points relate to 2D or 3D coordinates of the movement in space.16. The method of claim 10, wherein the indicator indicates a velocityof a movement associated with the gesture.
 17. A vehicle head unit of avehicle, the vehicle head unit comprising: a data interface configuredfor receiving gesture data, the gesture data including an indicatorindicating a gesture from a predefined set of gestures, the gesture dataincluding gestures detected based on image data from a 2d sensor of aportable device; and a portable device interface being configured for:establishing a data connection between the portable device and the datainterface, wherein the data interface is further configured forreceiving the gesture data via the data connection from a portabledevice application executed on the portable device; and a vehicleprocessing unit configured for controlling user interfacing with a userinterface based on the gesture indicated by the gesture data.
 18. Thevehicle head unit of claim 17, wherein the indicator indicates thegesture by specifying one or more of a start point, and end point, aseries of trace points, and a velocity of a movement associated with thegesture.
 19. The vehicle head unit of claim 17, wherein the fields ofview of the 2d sensor of the portable device and the at least one other2d sensor overlap in a common region located in free space of apassenger compartment between a middle console of the vehicle and acenter console of the vehicle.
 20. The vehicle head unit of claim 17,wherein detecting the gesture based on the image data from the 2d sensorof a portable device and the at least one other 2d sensor comprisesredundant and multiple recognizing of the detected gestures from imagedata from both the 2d sensor of the portable device and the at least oneother 2d sensor in a correlated manner.